Multimedia networked system detecting congestion by monitoring buffers&#39; threshold and compensating by reducing video transmittal rate then reducing audio playback rate

ABSTRACT

Disclosed is a networked multimedia information system which may be utilized to record, store and distribute multimedia presentations together with any supplemental materials that may be referenced during the presentation. The recorded presentation, together with the associated supplemental materials, may be simultaneously presented on a display containing two separate viewing windows. The effects of network congestion are minimized by prefetching audio and video data for storage in audio and video buffers. An adaptive control algorithm compensates for network congestion by dynamically varying the rate at which video frames are retrieved over the network, in response to network traffic conditions. The audio playback speed is reduced if the audio data does not arrive fast enough over the network to maintain the desired size of the audio buffer after the amount of video data transmitted across the network has been reduced to a minimum value.

This is a divisional of application Ser. No. 08/201,863 filed on Feb.24, 1994 now abandoned.

FIELD OF THE INVENTION

The present invention relates to a networked multimedia informationsystem, and more particularly, to a system for the storage anddistribution of a recorded multimedia presentation, such as a seminar orconference, together with any supplemental materials, such asviewgraphs, slides or handouts, that are referenced during thepresentation.

BACKGROUND OF THE INVENTION

Businesses today require access to ever-increasing amounts ofinformation, such as reports, correspondence, contracts, engineeringdrawings and marketing materials. Making this information available to alarge number of distributed corporate employees is often a difficultproblem.

There are a number of commercially available document managementsystems, such as NCR's Document Management System and Bellcore'sSuperBook Document Browser, which allow still images of documents,photographs and diagrams to be electronically stored for subsequentaccess over a network.

For example, NCR's Document Management System controls the flow of imagedocuments over a corporate network. A user employing a microcomputer orworkstation can access documents that have been stored in an electronicformat on a central file server using imaging technology.

However, there are few commercially available document managementsystems that also allow continuous media, such as recordings of audioand video presentations, to be stored and distributed over a network.Although multimedia applications make enormous demands on the resourcesof a computer system, recent advances in storage, processing,compression and network technologies have facilitated the development ofnetworked multimedia information systems. These technologies haveadvanced to where it is now possible to combine numerous media,including documents, audio and video, into a single system.

Since the number of employees who may participate in a corporatepresentation, such as a conference or a seminar, is often limited byscheduling and location constraints, it is desirable to provide a systemthat can record and store corporate presentations for subsequent viewingby additional corporate employees at convenient times and locations.

A major difficulty with subsequent viewing of recorded presentations,however, is the poor reproduction of supplemental materials associatedwith the recorded presentation, such as viewgraphs, slides and handouts.Accordingly, a need exists for an improved networked multimedia systemcapable of storing and indexing multimedia presentations, such asseminars and conferences, for subsequent access over a communicationsnetwork. A further need exists for a networked multimedia system thatseparately processes the supplemental materials, such as viewgraphs,slides and handouts, associated with a recorded presentation in order toimprove the clarity of the reproduced supplemental materials. Inaddition, a need exists for a networked multimedia system that iscapable of synchronizing the audio and video components of a recordedpresentation with the display of any supplemental materials referencedduring the presentation.

SUMMARY OF THE INVENTION

Generally, according to one aspect of the invention, a networkedmultimedia information system is utilized to store and distributemultimedia objects, such as recorded presentations, over a network to aplurality of workstations. The networked multimedia information systemincludes an information retrieval system, a still image storage andretrieval system and a continuous media storage and retrieval system.

According to one aspect of the invention, the networked multimediasystem is utilized to record, store and distribute multimediapresentations, such as seminars, meetings or conferences, and anysupplemental materials that may be referenced during the presentation,such as viewgraphs, slides, blackboard annotations or handouts.

According to another aspect of the invention, the recorded presentation,and the associated supplemental materials may be simultaneouslypresented on a display containing two separate viewing windows.Preferably, the presentation of the audio and video components of therecorded presentation is synchronized with the presentation of thesupplemental materials. In one embodiment, the synchronization of therecorded presentation and the associated supplemental materials isaccomplished by means of hyperlink files, which identify thesupplemental materials and the corresponding frame number in which thesupplemental materials are referenced.

According to a further feature of the invention, a user can manipulatethe supplemental materials displayed in one viewing window to controlthe video playback in the second viewing window, thereby allowing thesupplemental materials to serve as an index to the recordedpresentation. Once the user has selected the desired page or portion ofthe supplemental materials, the recorded presentation can be restartedfrom the video frame where the selected page or portion of thesupplemental materials is referenced in the recorded presentation.

Another feature of the invention minimizes the effects of networkcongestion by prefetching audio and video data for storage in audio andvideo buffers. In one embodiment, the networked multimedia systemcompensates for network congestion by using an adaptive controlalgorithm, which dynamically varies the rate at which video frames areretrieved from the respective file server over the network, in responseto network traffic conditions. In order to maximize the playback qualityof the recorded presentation, the audio component of the recordedpresentation is given preference over the video component.

Thus, when network congestion conditions are extreme, the networkmultimedia system will transmit only audio data, without any video data,to the respective workstation. If the audio data does not arrive fastenough over the network to maintain the desired size of the audio bufferwhen there is no video data being transmitted, the network multimediasystem will reduce the speed at which the audio data is played by theworkstation until the amount of audio data in the audio buffer hasreturned to the desired size.

A more complete understanding of the present invention may be had byreference to the following Detailed Description with the accompanyingdrawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic block diagram illustrating a networked multimediainformation system according to the present invention;

FIG. 2 is a schematic block diagram illustrating a still imageinformation capture subsystem;

FIG. 3 is a schematic block diagram illustrating a continuous mediainformation capture subsystem;

FIG. 4 illustrates an object profile that maintains bibliographicinformation for a multimedia object stored in the networked multimediasystem of FIG. 1;

FIG. 5 illustrates a workstation display having two windows forsimultaneous display of a recorded presentation, together with itsassociated supplemental materials;

FIG. 6 illustrates a still image hyperlink file that maintains thestarting video frame number associated with each page of the still imagesupplemental materials;

FIG. 7 illustrates a flow chart describing an exemplary still imagesynchronization subroutine according to the present invention asutilized by a known video process to synchronize the display of therecorded presentation with its still image supplemental materials on theworkstation display of FIG. 5;

FIG. 8 illustrates a continuous media hyperlink file that maintains anevent marker and the associated starting video frame number for eachblackboard event, or stroke, that is placed on an electronic blackboard;

FIG. 9 illustrates a flow chart describing an exemplary continuous mediasynchronization subroutine according to the present invention asutilized by a known video process to synchronize the display of therecorded presentation with its continuous media supplemental materialson the workstation display,of FIG. 5; and

FIG. 10 illustrates a flow chart describing an exemplary data buffermonitoring subroutine according to the present invention as utilized tomaintain a pre-defined amount of data in audio and video buffers.

DETAILED DESCRIPTION

The networked multimedia system 10 according to the present invention isillustrated in FIG. 1. The multimedia system 10 stores and indexesmultimedia objects, such as documents, diagrams, photographs, audio,video and animation, for access by a user over a network 20.

A user employs a workstation, such as workstation 15, which may be a SunSPARC workstation, model 10SX, 10M or classic M, produced by SunMicrosystems, Inc., of Mountain View, Calif., or a similar workstationoffering high-resolution imaging and support for real-time audio. Eachworkstation, such as the workstation 15, may be comprised of a display510, a processing unit 16, a memory unit 18, buses 22, audio hardware23, an audio buffer 110, a video buffer 115 and a user interface 24,such as a mouse and/or a keyboard. A preferred embodiment of the display510 is discussed below relative to FIG. 5.

The network 20 may be a local area network (LAN) or a wide area network(WAN), such as a nationwide corporate internet consisting of Ethernetnetworks interconnected by leased data circuits. Alternatively, thenetwork 20 may be embodied as, e.g., an asynchronous transfer mode (ATM)network or an Integrated Services Digital Network (ISDN). As discussedfurther below, to accommodate the low bandwidth which is typical ofthese networks, data files are preferably compressed prior totransmission over the network 20.

As illustrated in FIG. 1 and discussed further below, the networkedmultimedia system 10 utilizes an information retrieval system 50 tocoordinate the accessing of multimedia objects stored by a plurality offile servers, such as file servers 35, 40. The information retrievalsystem 50 will interact with a still image storage and retrieval system60 to retrieve the still image objects stored by an associated fileserver 35. In addition, the information retrieval system 50 willinteract with a continuous media storage and retrieval system 70 toretrieve the continuous media objects stored by an associated fileserver 40.

The information retrieval system 50 is preferably embodied as the LinusDatabase System, developed by AT&T, which is a "front end" system thatallows a workstation to access information over a network that is storedin a number of distributed databases. For a description of the LinusDatabase System, see Lorinda L. Cherry and Robert K. Waldstein,"Electronic Access to Full Document Text and Images through Linus," AT&TTechnical Journal, Vol. 68, No. 4, pp. 72-90 (July/August 1989),incorporated herein by reference.

An information retrieval system 50, such as the Linus Database System,typically provides the ability to store, index, search, and retrieveinformation from a plurality of databases. In addition, such informationretrieval systems 50 typically provide an authentication mechanism thatensures that stored information is viewed only by authorized users.

The still image storage and retrieval system 60 is preferably embodiedas the Ferret Document Browsing System, developed by AT&T, whichtogether with file server 35, provides storage and retrieval functionsfor a collection of databases, such as databases 80, 85, 90, containingstill images, such as images of documents, drawings and photographs. Fora description of the Ferret Document Browsing System, see Howard P.Katseff and Thomas B. London, "The Ferret Document Browser," Proc.USENIX Summer 1993 Technical Conference, Cincinnati, OH. (June 1993),incorporated herein by reference.

Alternatively, the information retrieval system 50 and the still imagestorage and retrieval system 60 may be embodied as a single integratedsystem, such as the Document Management System, commercially availablefrom NCR Corp., or the RightPages Image-Based Electronic Library System,developed by AT&T Bell Laboratories. For a description of the RightPagesImage-Based Electronic Library System, see Guy A. Story et al., "TheRightPages Image-Based Electronic Library for Alerting and Browsing,"Computer, 17-26 (September 1992); Lawrence O'Gorman, "Image and DocumentProcessing Techniques for the RightPages Electronic Library System,"Proc. 11th International Conference on Pattern Recognition, TheNetherlands, 260-63 (August 30-September 3, 1992), each incorporatedherein by reference.

The still image storage and retrieval system 60 preferably includes astill image information capture subsystem 210, discussed further belowrelative to FIG. 2, which is utilized to obtain an electronicrepresentation of each still image object and to store the electronicrepresentation on a file server, such as file server 35.

The still image objects are preferably stored by a still image fileserver, such as file server 35, on a plurality of disk arrays or othersecondary storage devices. The still image file server 35 may beembodied as a Sun SPARC workstation, commercially available from SunMicrosystems, Inc., of Mountain View, Calif. Alternatively, the stillimage objects may be stored on other mass storage devices that allowstored data to be accessed and shared by a plurality of workstations 15,25, 30 over a network 20. In a preferred embodiment, file server 35 canbe embodied as a plurality of distributed file servers which may bepositioned at a plurality of remote locations throughout network 20.

The continuous media storage and retrieval system 70, together with fileserver 40, provides storage and retrieval functions for a collection ofdatabases, such as databases 95, 100, containing continuous media, suchas video, audio and animated graphics. For a discussion of anillustrative continuous media storage and retrieval system 70, seeLawrence A. Rowe and Brian C. Smith, "A Continuous Media Player," Proc.3d Int'l Workshop on Network and Operating System Support for DigitalAudio and Video, San Diego, Calif. (Novenber 1992), incorporated hereinby reference. The Continuous Media Player system is available from theUniversity of California, Berkeley.

The continuous media storage and retrieval system 70, such as theContinuous Media Player system referenced above, preferably provides themechanism for digitizing and compressing audio, video and othercontinuous media for storage by a number of distributed file servers, ina manner described further below. In addition, the continuous mediastorage and retrieval system 70 provides the mechanism for accessingstored continuous media data files over network 20 for presentation on aworkstation, such as workstation 15.

Alternatively, the continuous media storage and retrieval system 70 maybe embodied as the multimedia network system commercially available fromFluent, Inc., now called Novell Multimedia.

The continuous media storage and retrieval system 70 preferably includesa continuous media information capture subsystem 310, discussed furtherbelow relative to FIG. 3, which is utilized to obtain an electronicrepresentation of each continuous media object and to store theelectronic representation on a file server, such as file server 40.

The continuous media objects are preferably stored by a continuous mediafile server, such as file server 40, on a plurality of disk arrays orother secondary storage devices. For a discussion of illustrativecontinuous media file servers, see Roger L. Haskin, "The SharkContinuous-Media File Server," Proc. IEEE COMPCON '93, pp. 12-15, SanFrancisco, Calif. (February 1993); Fouad A. Tobagi & Joseph Pang,"StarWorks™--A Video Applications Server," Proc. IEEE COMPCON '93, pp.4-11, San Francisco, Calif. (February 1993). In a preferred embodiment,file server 40 can be embodied as a plurality of distributed fileservers which may be positioned at a plurality of remote locationsthroughout network 20.

In a preferred embodiment, each multimedia object stored in themultimedia system 10 is indexed by a content profiler 75 that acquiresbibliographic information on each multimedia object for storage in aprofile database 105, as discussed further below.

As illustrated in FIGS. 2 and 3, multimedia objects are stored by thenetworked multimedia system 10 by capture processes that obtain anelectronic representation of each multimedia object. Multimediainformation that is received in an electronic format, such as a documentreceived directly from a computing system in electronic form, may beentered directly in the system 10. Multimedia information that isreceived in a non-electronic format, however, such as informationreceived on paper or film, must first be converted into an electronicformat using imaging technology.

An illustrative still image information capture subsystem 210 for use bythe still image storage and retrieval system 60 is shown in FIG. 2. Thestill image information capture subsystem 210 will convert a printeddocument 215 or diagram 220 into an electronic format using a documentscanner 225, which will convert a printed page 215, 220 into anelectronic format by making a bitmap image of the page. Similarly, aphotograph 230 can preferably be converted into an electronic format forentry into the system 10 using a grayscale scanner 235.

The electronic format images produced by scanner 225 and grayscalescanner 235 preferably conform to the well-known Tag Image File Format(TIFF) specification. The scanned images are preferably compressed bycompression circuitry 245, 250, respectively, prior to storage by fileserver 35.

Multimedia information that is received in an electronic form, such asdocuments that are received directly from a source computer 255 inelectronic form, can be stored directly by file server 35 followingcompression by compression circuitry 260.

The textual portions of a document 215 or a diagram 220 are preferablyconverted into a computer-readable format by an optical characterrecognition system 240 for transfer to the content profiler 75 via datalink 270. Similarly, a copy of each document received directly from asource computer 255 in electronic computer-readable form is alsopreferably transferred to the content profiler 75, via data link 275. Asdiscussed further below, the computer-readable form of a multimediaobject may be utilized by the content profiler 75 for the automaticgeneration of an object profile, which may consist of an abstract andother bibliographic information.

An illustrative continuous media information capture subsystem 310 foruse by the continuous media storage and retrieval system 70 is shown inFIG. 3.

An analog video input module 315 is preferably configured to receiveanalog video signals via data link 320 from, e.g., a video cassetterecorder 325 and/or a video camera 330, as shown in FIG. 3. Similarly,an analog audio input module 340 is preferably configured to receiveanalog audio signals via data link 345, from, e.g., an audio taperecorder 350 and/or a microphone 355. Video camera 330 and microphone355 may be positioned during a presentation, such as a meeting orseminar, to capture the visual and audio portions of the presentation,respectively.

In this manner, analog video and audio signals generated during apresentation by camera 330 and/or microphone 355, respectively, may bereceived by input modules 315, 340 for storage in the multimedia system10.

Preferably, the analog video and audio signals received by input modules315, 340 are processed by a digitizer/compressor 360, prior to storageby the file server 40. The digitizer/compressor 360 may be embodied as aParallax XVideo System, consisting of a board having a JPEG CODEC chipand associated software, commercially available from Parallax Graphics,Inc. of Santa Clara, Calif. The digitizer/compressor 360 will receiveanalog video and audio inputs from input modules 315, 340. Thedigitizer/compressor 360 will preferably digitize and compress thereceived video input and digitize the received audio input.

The data stream generated by the digitizer/compressor 360 is preferablyin the JPEG Movie File format, which is described in "JPEG Movie FileSpecification, Release 1.0," Parallax Graphics, Inc., Santa Clara,Calif. (Nov. 5, 1992), incorporated herein by reference. The JPEG MovieFile format interleaves one frames' worth of audio with a frame ofvideo. For example, if video signals are being stored for a display rateof 10 frames per second (fps), the resulting interleaved data file willinclude a repeating pattern of one frame of video, followed by a tenthof a second's worth of audio.

The JPEG Movie File format is preferred because of the inherentsynchronization of the audio and video streams that results from thepaired storage of a video frame with its associated audio, as discussedfurther below. In addition, since the JPEG technique compresses eachvideo frame independently, there is a resulting frame independence thatallows a video frame to be easily dropped in the event of networkcongestion, as discussed below. Other video compression schemes may beutilized, such as the MPEG compression standard. However, the MPEGcompression technique, which takes into account the similarity betweensuccessive images, is less preferred because the resulting framedependence makes it more difficult to drop video frames in the event ofnetwork congestion.

It is noted that typical workstations, such as the Sun workstationsdiscussed above, include audio hardware 23 for receiving analog audioinputs and storing the audio signals in a digitized format. Thus, whereanalog audio signals are being processed, without any associated video,a workstation, such as a Sun workstation, can receive the analog audiosignal via its audio input port, and then digitize the audio signal forstorage as a multimedia object by a file server, such as the file server40.

An electronic blackboard 358 is preferably provided, so that annotationsmade by a speaker on the blackboard 358 during a presentation may beelectronically captured for storage in the multimedia information system10. The digital signals generated by the electronic blackboard 358 canbe stored directly by file server 40 in an electronic file followingcompression by compression circuitry 372.

A digital audio/video input module 370 is preferably configured toreceive digital signals from a plurality of digital audio or videosources, such as a source 365, which may be a digital videodisk player.The digital signals received by input module 370 can be stored directlyby file server 40 following compression by compression circuitry 375.

Other continuous multimedia information that is received in anelectronic form directly from a source computer 380, such as animatedgraphics, can be stored by file server 40 following compression bycompression circuitry 385.

Once the multimedia information has been stored in an electronic formatby the information capture subsystems 210, 310 in the manner describedabove, the resulting multimedia object may be selected by a user andthen transmitted over the network 20 to a workstation 15. A workstation,such as the workstation 15, will receive the transmitted multimediaobject for presentation to the user.

In a preferred embodiment, the content profiler 75 will create an objectprofile 400, shown illustratively in FIG. 4, for each multimedia object.Each object profile 400 includes bibliographic information that may beused for indexing and searching each multimedia object. In a preferredembodiment, each object profile 400 is stored centrally in a profiledatabase 105 associated with the content profiler 75, as shown in FIG.1.

The object profile 400 can include a plurality of entries for listingbibliographic information that describes relevant aspects of eachmultimedia object, such as a title entry 410, an author or speaker entry420, an abstract entry 430, a date entry 440 and a keyword entry 450 forlisting the keywords associated with the multimedia object.

In addition, each object profile 400 preferably includes an entry 460containing a pointer to the memory location where the multimedia objectis stored, i.e., an indication of the particular file server, such asfile server 35 or 40, which controls the storage of the multimediaobject. In one embodiment of the invention, discussed further below,entry 460 can be a multi-valued entry containing pointers to a number ofrelated multimedia objects and/or data files.

As indicated above, the content profiler 75 can automatically generateobject profiles, such as the object profile 400, for those multimediaobjects that are stored in a computer-readable form. The contentprofiler 75 will receive multimedia objects in a computer-readable formvia data link 270 from the optical character recognition system 240 andvia data link 275 from source computer 255, as illustrated in FIG. 2.The object profiles associated with multimedia objects that are notstored in a computer-readable form, however, such as the object profilesfor photographs, audio and video, can be generated manually for entry inthe content profiler 75.

The information retrieval system 50 allows a user to access themultimedia information that has been stored in the plurality ofdatabases, such as databases 80, 85, 90, 95, 100, associated with fileservers 35, 40. As discussed above, each multimedia object has anassociated object profile, such as the object profile 400, that isstored in a profile database 105. The information retrieval system 50preferably provides a mechanism for searching the plurality of objectprofiles stored in the profile database 105 to identify those multimediaobjects that satisfy a user-specified query.

A user may employ a workstation, such as the workstation 15, to enter aquery specifying parameters for retrieving multimedia objects from themultimedia system 10. The information retrieval system 50 will retrieveall the object profiles from the profile database 105 that correspond tomultimedia objects that satisfy the user-specified query. Thereafter,the information retrieval system 50 will transmit the object profilessatisfying the entered query to the workstation 15. The user can reviewthose object profiles that satisfy the entered query and select thosemultimedia objects that should be downloaded to the workstation 15.

Once a multimedia object has been selected by a user, the informationretrieval system 50 will access the entry 460 in the object profile thatindicates the memory locations where the selected multimedia objects arestored. The information retrieval system 50 will send a retrievalmessage over network 20 to the respective storage and retrieval system60, 70 which accesses the appropriate file server 35, 40 indicated bythe retrieved pointer. The compressed file containing the requestedmultimedia object is transmitted from the respective file server to theworkstation 15, in a known manner.

If the retrieved multimedia object is a still image object with manypages, such as a document, the workstation 15 will thereafter decompresseach page of the multimedia object as requested for display by the user,in a known manner.

If, however, the retrieved multimedia object is comprised of continuousmedia, such as audio or video, the entire multimedia object will notnormally be retrieved at once, due to the memory limitations associatedwith many workstations. Rather, the workstation 15 will utilize aprefetch routine to retrieve several frames worth of audio and videofrom the respective file server for storage in an audio buffer 110 and avideo buffer 115, respectively, as described further below. Accordingly,the received video frames will be decompressed by the workstation 15 asthey are received over the network 20 from the respective file server,in a known manner.

For a multimedia object that contains a plurality of media, such as theaudio and video portions of a recorded presentation, the workstation 15will preferably have a mechanism for synchronizing the presentation ofthe various outputs. In the preferred embodiment discussed above, amultimedia object containing the audio and video portions of a recordedpresentation will be stored as a data file containing interleaved audioand video information in the JPEG Movie File format.

The workstation 15 can process the interleaved JPEG data file, having aframe of video paired with its associated audio, and direct the audioinformation, in real time, to the audio hardware 23 of the workstation15, in a known manner, while the video information is decompressedutilizing a software routine for presentation on display 510, in a knownmanner.

Alternatively, the multimedia system 10 can implement knownsynchronization algorithms to control the recording and playback ofcontinuous media. During playback, the synchronization algorithmmonitors the internal clock of the workstation, such as workstation 15,to determine which audio and video frames should be played and makesadjustments, if necessary.

According to one aspect of the invention, the above networked multimediasystem 10 is utilized to record and store multimedia presentations, suchas seminars, meetings or conferences, together with any supplementalmaterials that may be referenced during the presentation, such asviewgraphs, slides, handouts or blackboard annotations. The recordedpresentation, together with the stored supplemental materials, maythereafter be accessed over network 20 for presentation to a user on aworkstation, such as the workstation 15.

The audio and video portions of the presentation are preferably capturedand converted into a multimedia object using the continuous mediainformation capture subsystem 310, in the manner described aboverelative to FIG. 3. The supplemental materials that consist of stillimages, i.e., viewgraphs, slides and handouts, are preferably separatelyprocessed from the recorded presentation and stored as a separatemultimedia object by the still image information capture subsystem 210,in the manner described above relative to FIG. 2.

In addition, according to a further feature of the invention discussedfurther below, the supplemental materials that consist of continuousmedia, i.e., the annotations made by a speaker on an electronicblackboard 358, are preferably stored as a separate multimedia object bythe continuous media information capture subsystem 310, in the mannerdescribed above relative to FIG. 3.

In a preferred embodiment, illustrated in FIG. 5, the recordedpresentation, together with the stored supplemental materials, may besimultaneously presented on a display, such as display 510 ofworkstation 15, containing two separate viewing windows 520, 530. Inthis manner, the appropriate supplemental materials may be displayed inwindow 520 as they are referenced during the recorded presentation beingdisplayed in window 530.

Preferably, the presentation of the audio and video components of therecorded presentation is synchronized with the presentation of thesupplemental materials in window 520. As discussed further below, thesupplemental materials are an additional medium to be synchronized withthe presentation of the audio and video portions of the recordedpresentation.

As is known, a video presentation consists of a large number ofsequential video frames which may be recorded at a fixed rate, such as aframe rate of 10 frames per second. Thus, the video frame number can beutilized as an index into the video stream. In addition, as discussedfurther below, the supplemental materials that are associated with therecorded presentation, such as the viewgraphs, may be identified withthe video frames in which they are referenced.

SYNCHRONIZATION OF THE RECORDED PRESENTATION AND STILL IMAGESUPPLEMENTAL MATERIALS

Where the supplemental materials associated with the recordedpresentation is comprised of a multi-page still image object, such asviewgraphs and handouts, each page of the still image multimedia objectmay be numbered sequentially in the order in which it is referencedduring the recorded presentation. The recorded presentation and theassociated still image supplemental materials may thereafter besynchronized by means of a still image hyperlink file, such as the stillimage hyperlink file 600 illustrated in FIG. 6.

The still image hyperlink file 600 preferably consists of a plurality ofvideo frame-still image page pairs, i.e., the page number of the stillimage multimedia object in column 620 together with the correspondingframe number in column 610 in which the still image page is firstreferenced.

The still image hyperlink file 600 will include a plurality of rows,such as rows 625, 630, 635, 640, 642, 644, 646, each corresponding to avideo frame-still image page pair. Each time a page of the still imagesupplemental material object, such as a viewgraph, is referenced duringthe presentation, an entry is made in column 620 of the still imagehyperlink file 600 with the appropriate page number of the supplementalmaterial, together with the number in column 610 of the video frame inwhich the supplemental material page is first referenced. If the sameviewgraph, or other still image supplemental material, is referencedmore than once during a recorded presentation, it is preferably assigneda unique page number for each appearance of the still image page.

According to a feature of the invention, a pair of independentprocesses, which may interact by means of a message passing mechanism,are preferably utilized to display a recorded presentation and itsassociated still image supplemental materials. A video process controlsthe presentation of the audio and video portions of the recordedpresentation. The video process preferably incorporates the features andfunctions of the presentation processes of the Continuous Media Player,described in the reference incorporated above, as well as the additionalfeatures and functions described below.

Where the recorded presentation has associated still image supplementalmaterials, the video process will interact with a still imagepresentation process in order to present the appropriate still imagesupplemental materials in the window 520 of display 510. The still imagepresentation process preferably incorporates the features and functionsof the still image presentation processes of the Ferret DocumentBrowser, described in the reference incorporated above, as well as theadditional features and functions described below.

The video process preferably utilizes a still image synchronizationsubroutine, illustrated in FIG. 7, to synchronize the still imagesupplemental materials in window 520 with the ongoing video presentationin window 530. If the recorded presentation has associated still imagesupplemental materials, the video process will enter the still imagesynchronization subroutine at step 700 as each video frame is presentedin window 530, as shown in FIG. 7.

During step 710, the still image synchronization subroutine will accessthe still image hyperlink file 600 to determine if the appropriate pageof the still image file is being displayed in window 520. The stillimage synchronization subroutine will access the row of the still imagehyperlink file 600 having the largest video frame number that is lessthan or equal to the frame number currently being displayed.

Once the appropriate row of the still image hyperlink file 600 isaccessed, the still image synchronization subroutine will retrieve,during step 720, the appropriate still image page number from thecorresponding entry in column 620. The still image synchronizationsubroutine will compare during step 730 the still image page numberretrieved from the still image hyperlink file 600 in the previous stepto the page number of the still image currently displayed. If it isdetermined during step 730 that the retrieved still image page numberdoes equal the currently displayed still image page number, theappropriate still image is being displayed, and the still imagesynchronization subroutine will be exited at step 760. Process controlwill thereafter return to the video process.

If, however, it is determined during step 730 that the retrieved stillimage page number does not equal the currently displayed still imagepage number, the synchronization subroutine will send a message to thestill image presentation process during step 750, which includes anindication of the still image page number which should be displayed inwindow 520, as retrieved during step 720. The still imagesynchronization subroutine will thereafter be exited during step 760 andprocess control will return to the video process.

When the still image presentation process receives the still image pagenumber from the still image synchronization subroutine, transmittedduring step 750, the still image presentation process will decompressthe appropriate page of the supplemental materials multimedia objectcorresponding to the received still image page number for presentationin window 520, in a known manner.

SYNCHRONIZATION OF THE RECORDED PRESENTATION AND CONTINUOUS MEDIASUPPLEMENTAL MATERIALS

According to a further feature of the invention, the supplementalmaterials associated with a recorded presentation may consist ofcontinuous media, such as the annotations made by a speaker on anelectronic blackboard 358 during the associated presentation.Preferably, the electronic file that stores the blackboard annotationsor other continuous media supplemental materials associated with arecorded presentation is marked with a time stamp or other counter whichmay serve as an index into the recorded blackboard annotations. In thismanner, recorded blackboard events, i.e., the placement of strokes onthe electronic blackboard 358, can be presented with the video frames inwhich the blackboard event is referenced.

The recorded presentation and associated continuous media supplementalmaterials are preferably synchronized by means of a continuous mediahyperlink file, such as the continuous media hyperlink file 800illustrated in FIG. 8. The continuous media hyperlink file 800preferably consists of a plurality of video. frame-event marker pairs,i.e., a marker that identifies the position of the recorded blackboardevent within the associated electronic file in column 820 together withthe corresponding frame number in column 810 in which the correspondingblackboard event is first referenced.

The continuous media hyperlink file 800 will include a plurality ofrows, such as rows 830, 840, 850, 860, 870, each corresponding to avideo frame-event marker pair. For each blackboard event, or stroke,that is placed on the electronic blackboard 358 during the presentation,an entry is preferably made in column 820 of the continuous mediahyperlink file 800 with the event marker that identifies the position ofthe recorded blackboard event within the associated supplementalmaterial electronic file. In addition, the video frame number in whichthe blackboard event is first referenced is placed in column 810.

When the recorded presentation has associated continuous mediasupplemental materials, the video process, discussed above, willpreferably interact with a continuous media presentation process inorder to coordinate the presentation of the appropriate continuous mediasupplemental materials in the window 520 of display 510. The continuousmedia presentation process preferably incorporates the features andfunctions of the presentation processes of the Continuous Media Player,in a manner similar to the video process, as well as the additionalfeatures and functions described below.

The video process preferably utilizes a continuous media synchronizationsubroutine, illustrated in FIG. 9, to synchronize the continuous mediasupplemental materials in window 520 with the ongoing video presentationin window 530. If the recorded presentation has associated continuousmedia supplemental materials, the video process will enter thecontinuous media synchronization subroutine at step 900 as each videoframe is presented in window 530, as shown in FIG. 9.

During step 910, the continuous media synchronization subroutine willaccess the continuous media hyperlink file 800 to determine if theappropriate segment of the electronic file, i.e., the appropriateblackboard event, is being presented in window 520. The continuous mediasynchronization subroutine will access the row of the continuous mediahyperlink file 800 having the largest video frame number that is lessthan or equal to the frame number currently being displayed.

Once the appropriate row of the continuous media hyperlink file 800 isaccessed, the continuous media synchronization subroutine will retrieve,during step 920, the appropriate event marker from the correspondingentry in column 820. During step 930, the continuous mediasynchronization subroutine will compare the event marker retrieved fromthe continuous media hyperlink file 800 during step 920 to the eventmarker associated with the blackboard event currently being displayed.

If it is determined during step 930 that the retrieved event markerequals the event marker associated with the blackboard event currentlybeing displayed, the appropriate blackboard event is being displayed,and the continuous media synchronization subroutine will be exited atstep 960. Process control will thereafter return to the video process.

If, however, it is determined during step 930 that the retrieved eventmarker does not equal the event marker associated with the blackboardevent currently being displayed, the continuous media synchronizationsubroutine will send a message to the continuous media presentationprocess during step 950, which includes the event marker as anindication of the file position from which the associated electronicfile that contains the recorded blackboard annotations should berestarted. The continuous media synchronization subroutine willthereafter be exited during step 960 and process control will return tothe video process.

When the continuous media presentation process receives the event markerfrom the continuous media synchronization subroutine, transmitted duringstep 950, the continuous media presentation process will make anadjustment to the presentation of the continuous media supplementalmaterials in window 520 by restarting the presentation from the positionin the electronic file indicated by the received event marker, in aknown manner.

When a recorded presentation has associated still image and/orcontinuous media supplemental materials, a single object profile,similar to the object profile 400 illustrated in FIG. 4, is preferablyutilized to profile the multimedia objects corresponding to the recordedpresentation and associated supplemental materials. The entry 460 ofobject profile 400 is preferably a multi-valued entry containing aplurality of pointers, e.g., a pointer to the multimedia objectcorresponding to the recorded presentation, a pointer to the multimediaobject corresponding to the supplemental materials and, whereappropriate, a pointer to the hyperlink file 600 associated with therecorded presentation.

Thus, when a user selects a recorded presentation having associatedsupplemental materials, in the manner described above, the informationretrieval system 50 will access the entry 460 to retrieve the pluralityof pointers. Thereafter, the information retrieval system 50 will send aretrieval message over network 20 to each file server indicated by theplurality of retrieved pointers. The data files containing the recordedpresentation, corresponding supplemental materials, and, whereappropriate, the respective hyperlink file 600, 800, will be transmittedby the respective file servers to the workstation, such as workstation15.

Window 530 of display 510 includes a frame number scroll bar 540 thatallows a user to scroll through video frames. Once the user has selecteda starting frame using the frame number scroll bar 540, the videoprocess will restart the video, together with the corresponding audio,from the selected video frame. If the recorded presentation hasassociated still image supplemental materials, the video process willutilize the still image synchronization subroutine of FIG. 7 to retrievethe appropriate still image page number from the still image hyperlinkfile 600 corresponding to the selected starting video frame, fortransmittal to the still image presentation process.

Similarly, if the recorded presentation has associated continuous mediasupplemental materials, the video process will utilize the continuousmedia synchronization subroutine of FIG. 9 to retrieve the appropriateevent marker from the continuous media hyperlink file 800 correspondingto the starting video frame selected using the frame number scroll bar540, for transmittal to the continuous media presentation process.

It is noted that if a user has selected a frame number that has not yetbeen received by the workstation 15 from the respective file server, thevideo process must first transmit a retrieval request to the file serverto retrieve the necessary frames.

In addition, window 530 includes a playback speed scroll bar 550 thatallows a user to control the playback speed, in frames per second, ofthe recorded presentation, in a known manner. The playback speed scrollbar allows the user to adjust the playback speed from a minimum of zeroframes per second, i.e., a still image, up to the maximum recorded framerate of the video, in either forward or reverse mode. Once the user hasselected a video playback speed using the playback speed scroll bar 550,the video process will adjust the rate of data being requested from thestorage and retrieval system 70 to the selected playback speed, inaddition to making local adjustments to the video and audio outputs ofworkstation 15.

It is noted that where the recorded presentation has associated stillimage supplemental materials the video process will utilize the stillimage synchronization subroutine of FIG. 7 to retrieve the appropriatestill image page number from the still image hyperlink file 600 in thesame manner for both forward and reverse playback of the video, asdescribed above. Similarly, where the recorded presentation hasassociated continuous media supplemental materials, the video processwill utilize the continuous media synchronization subroutine of FIG. 9to retrieve the appropriate event marker from the continuous mediahyperlink file 800 in the same manner for both forward and reverseplayback of the video, as described above.

In a preferred embodiment, a user can manipulate the still image orcontinuous media supplemental materials in window 520 to control thevideo playback in window 530. In this manner, the presentation of thesupplemental materials can serve as an index to the recordedpresentation. A user can proceed through the supplemental materials inwindow 520, e.g., by clicking the left button of a mouse in window 520to move through the supplemental materials in a forward direction orclicking the right button of the mouse to move in a reverse direction.

Once the user has selected a desired still image page or continuousmedia event in window 520, the recorded presentation in window 530 canbe restarted from the video frame where the still image page orcontinuous media event is referenced in the recorded presentation. Uponselection of a desired page or event in window 520, the still imagepresentation process or the continuous media presentation process, asappropriate, preferably sends a message to the video process with theselected still image page number or continuous media event marker. Thevideo process will thereafter access the respective hyperlink file 600,800 to retrieve the video frame that corresponds to the selected stillimage page or continuous media event. The video process will thenrestart the video from the retrieved frame number.

It is noted that where the supplemental materials associated with arecorded presentation consist of a combination of still images andcontinuous media, a third viewing window (not shown) may be provided ondisplay 510 for the simultaneous presentation of the recordedpresentation, together with the stored still image supplementalmaterials and continuous media supplemental materials. Alternatively,window 520 may be configured to alternately present still images orcontinuous media, as they are referenced during the recordedpresentation, in a known manner.

NETWORK CONGESTION

In order to compensate for congestion on network 20 that causes thedelayed arrival of audio and video data, the video process willpreferably prefetch audio and video frames from the respective fileserver 40 for storage in audio and video buffers 110, 115, respectively,on the user's workstation to minimize the effect of network congestionon the playback of the recorded presentation. Since two-way videoconferencing applications demand rapid response times, the buffering ofaudio and video data would not normally be tolerated in suchapplications. It is noted, however, that during the playback of recordedpresentations the buffering of audio and video data would not bedetected by a user.

According to one feature of the invention, the networked multimediasystem compensates for congestion on network 20 using an adaptivecontrol algorithm to dynamically vary the rate at which video frames areretrieved from the respective file server 40 over network 20, inresponse to increases or decreases in the amount of other data beingtransmitted across the network 20. For a discussion of an illustrativeadaptive control algorithm, see section 2.3 of the Continuous MediaPlayer reference, incorporated above.

In order to maximize the playback quality of the recorded presentation,the audio component of the recorded presentation is preferably givenpreference over the video component. It has been found that a personviewing a recorded presentation will object more strongly to defects inthe audio portion of the recorded presentation than to defects in thevideo portions of the presentation.

In a preferred embodiment, the video process utilizes a data buffermonitoring subroutine, illustrated in FIG. 10, to maintain a pre-definedamount of audio and video data in the audio and video buffers 110, 115.The data buffer monitoring subroutine will continuously monitor theaudio and video buffers 110, 115 during step 1000, until the amount ofaudio or video data stored in the audio or video buffers 110, 115 dropsbelow a predefined threshold value, as detected by step 1010.

If it is determined during step 1010 that the amount of audio or videodata in the audio or video buffers 110, 115, respectively, has fallenbelow the desired threshold value, the data is not arriving at theworkstation 15 from the respective file server 40 over network 20 asfast as it is being presented to a user by the workstation 15. Thus, thedata buffer monitoring subroutine will reduce the requested videoplayback rate during step 1015 by requesting that the file server 40transmit fewer video frames per second to the workstation 15. It isnoted, however, that the data buffer monitoring subroutine willpreferably continue to request all of the audio data from the fileserver 40.

After the requested video playback rate is reduced during step 1015, atest is performed during step 1020 to determine if the amount of audioor video data in the audio or video buffers 110, 115 is still below thedesired threshold value. If it is determined during step 1020 that theamount of audio or video data in the audio or video buffers 110, 115 hasrisen above the desired threshold value, program control will proceed tostep 1060, discussed below.

If, however, it is determined during step 1020 that the amount of audioor video data in the audio or video buffers 110, 115 is still below thedesired threshold value, a test is performed during step 1025 todetermine if the requested video playback rate has been reduced to theminimum value, i.e., a playback rate of 0 frames per second (fps).

If it is determined during step 1025 that the requested video playbackrate has not yet been reduced to the minimum value, program control willreturn to step 1015 for a further reduction in the requested videoplayback rate.

If it is determined during step 1025 that the requested video playbackrate has been reduced to the minimum value, network congestionconditions are so extreme that even though no video data is beingtransmitted across the network 20, the audio data is still not arrivingfast enough over the network 20 to maintain the desired size of theaudio buffer 110. In a preferred embodiment, the data buffer monitoringsubroutine will compensate for the delayed arrival of audio data byplaying the audio data from the audio buffer 110 at slower thanreal-time.

Thus, the data buffer monitoring subroutine will begin playing the audioat a reduced speed during step 1030. The data buffer monitoringsubroutine will continue playing the audio at the reduced speed until itis determined during step 1040 that the amount of audio data in thebuffer has returned to the desired threshold value. Once it isdetermined during step 1040 that the amount of audio data in the bufferis greater than or equal to the desired threshold value, the data buffermonitoring subroutine will resume playing the audio at a normal, orreal-time, speed during step 1045. Thereafter, program control willreturn to step 1000, and continue in the manner described above.

The data buffer monitoring subroutine could play the audio at half-speedduring step 1030, e.g., by dividing each frames' worth of buffered audiodata into n segments and then playing each segment twice. If it isdesired to play the audio at a speed between half speed and normalspeed, not all of the n segments are played twice. Similarly, if it isdesired to play the audio at a speed less than half speed, some of the nsegments may be played more than twice.

In a preferred embodiment, the data buffer monitoring subroutine willgradually adjust the playback speeds of the audio during steps 1030 and1045 in order to make the transition from one speed to another lessnoticeable to a listener. For example, the data buffer monitoringsubroutine can reduce the audio playback rate during step 1030 accordingto a scale that gradually adjusts the playback rate between a definedmaximum and minimum audio playback rate. In addition, by monitoring therate at which the audio buffer 110 is emptying, the data buffermonitoring subroutine can determine how quickly the audio playback rateshould be reduced during step 1030 and what the minimum audio playbackrate should ultimately be.

In an alternate embodiment, the data buffer monitoring subroutine couldplay the audio at a reduced speed during step 1030 by utilizing awell-known pitch extraction process, which identifies where the pausesare in the audio and makes the pauses longer.

Once the data buffer monitoring subroutine has stabilized the amount ofdata in the audio and video buffers 110, 115, as detected during step1020, program control will proceed to step 1060. During step 1060, theamount of video data requested from the respective file server will begradually increased. Thereafter, program control will return to step1000 to continue monitoring the audio and video buffers 110, 115 in themanner described above.

Although the data buffer monitoring subroutine will drop video framesduring times of network congestion, it is preferred that the videoprocess will access the still image synchronization subroutine of FIG. 7where the supplemental materials consist of still images or thecontinuous media synchronization subroutine of FIG. 9 where thesupplemental materials consist of continuous media for each video frame,regardless of whether or not the video frame is actually displayed inwindow 530. In this manner, synchronization is maintained between thepresentation of the supplemental materials in window 520 and thecontinuing audio presentation, even in the absence of video frames.

It is to be understood that the embodiments and variations shown anddescribed herein are illustrative of the principles of this inventiononly and that various modifications may be implemented by those skilledin the art without departing from the scope and spirit of the invention.

We claim:
 1. A method for use by a computing system receiving audio andvideo data over a network for presentation to a user, said methodcompensating for congestion on said network which causes delayed arrivalof said audio and video data, said video data being transmitted oversaid network to said computing system at a requested video transmittalrate, said audio data being presented to said user at an audio playbackrate, said network congestion compensation method comprising the stepsof:maintaining an audio buffer for storing a predefined amount of saidaudio data received over said network; maintaining a video buffer forstoring a predefined amount of said video data received over saidnetwork; monitoring said audio and video buffers to determine when saidamount of audio or video data in said buffer falls below said predefinedamounts; reducing said requested video transmittal rate if saidmonitoring step determines that said amount of audio or video data insaid buffers has fallen below said predefined amounts; and reducing saidaudio playback rate if said amount of audio data in said audio buffer isbelow said predefined amount of audio data after said step of reducingsaid requested video transmittal rate.
 2. The network congestioncompensation method according to claim 1, further including the step ofincreasing said audio playback rate once said amount of audio data insaid audio buffer is above said predefined amount of audio data.
 3. Thenetwork congestion compensation method according to claim 1, whereinsaid step of reducing said audio playback rate is not performed unlesssaid step of reducing said requested video transmittal rate has reducedsaid requested video transmittal rate to a minimum value.
 4. The networkcongestion compensation method according to claim 1, wherein said stepof reducing said audio playback rate plays said audio at a reduced speedby dividing said audio data in said audio buffer into a plurality ofsegments and playing one or more of said segments at least twice.
 5. Thenetwork congestion compensation method according to claim 1, whereinsaid step of reducing said audio playback rate plays said audio at areduced speed by utilizing a pitch extraction process.
 6. The networkcongestion compensation method according to claim 1, wherein said stepof reducing said audio playback rate will gradually adjust said audioplayback rate to make the transition from one speed to another lessnoticeable to said user.
 7. A computer-readable storage mediumcomprising encoded computer-readable program instructions for use inconjunction with a programmable computer receiving audio and video dataover a network for presentation to a user, which instructions cause thecomputer to compensate for congestion on the network that results indelayed arrival of the audio and video data, the video data beingtransmitted over the network at a requested video transmittal rate to avideo buffer for storing a predefined amount of the video data and theaudio data being presented to the user at an audio playback rate,wherein a predefined amount of the audio data is stored in an audiobuffer, the program instructions defining steps to be performed by theprogrammable computer, the steps comprising:monitoring the audio andvideo buffers to determine when the amount of audio or video data in thebuffers falls below the predefined amounts; reducing the requested videotransmittal rate if the monitoring step determines that the amount ofaudio or video data in the buffers has fallen below the predefinedamounts; and reducing the audio playback rate if the amount of audiodata in the audio buffer is below the predefined amount of audio dataafter the step of reducing the requested video transmittal rate.
 8. Thecomputer-readable storage medium of claim 7, wherein the steps furthercomprise increasing the audio playback rate once the amount of audiodata in the audio buffer is above the predefined amount of audio data.9. The computer-readable storage medium of claim 7, wherein the step ofreducing the audio playback rate is not performed unless the step ofreducing the requested video transmittal rate has reduced the requestedvideo transmittal rate to a minimum value.
 10. The computer-readablestorage medium of claim 7, wherein the step of reducing the audioplayback rate plays the audio at a reduced speed by dividing the audiodata in the audio buffer into a plurality of segments and playing one ormore of the segments at least twice.
 11. The computer-readable storagemedium of claim 7, wherein the step of reducing the audio playback rateplays the audio at a reduced speed by utilizing a pitch extractionprocess.
 12. The computer-readable storage medium of claim 7, whereinthe step of reducing the audio playback rate will gradually adjust theaudio playback rate to make the transition from one speed to anotherless noticeable to the user.